A closer look at quality judgments of spoken dialog systems
نویسندگان
چکیده
User judgments of Spoken Dialog Systems provide evaluators of such systems with a valid measure of their overall quality. Models for the automatic prediction of user judgments have been built, following the introduction of PARADISE [1]. Main applications are the comparison of systems, the analysis of parameters affecting quality, and the adoption of dialog management strategies. However, a common model which applies to different systems and users has not been found so far. With the aim of getting a closer insight into the qualityrelevant characteristics of spoken interactions, an experiment was conducted where 25 users judged the same 5 dialogs. User judgments were collected after each dialog turn. The paper presents an analysis of the obtained results and some conclusions for future work.
منابع مشابه
Modeling User Satisfaction with Hidden Markov Models
Models for predicting judgments about the quality of Spoken Dialog Systems have been used as overall evaluation metric or as optimization functions in adaptive systems. We describe a new approach to such models, using Hidden Markov Models (HMMs). The user’s opinion is regarded as a continuous process evolving over time. We present the data collection method and results achieved with the HMM model.
متن کاملCorrelation Between Model-based Approximations of Grounding-related Cognition and User Judgments
As spoken dialog systems become more complex, efficient ways to evaluate them in early development stages are required. User simulation has been successfully used for this purpose. While current user models describe behavior on the level of overt behavior, modeling aspects of cognition can reveal direct insights into usability problems. Thus, in this paper we propose two models related to groun...
متن کاملEvaluating a Trainable Sentence Planner for a Spoken Dialogue System
Techniques for automatically training modules of a natural language generator have recently been proposed, but a fundamental concern is whether the quality of utterances produced with trainable components can compete with hand-crafted template-based or rulebased approaches. In this paper We experimentally evaluate a trainable sentence planner for a spoken dialogue system by eliciting subjective...
متن کاملTowards generic quality prediction models for spoken dialogue systems - a case study
In this paper, models are investigated which aim at predicting quality perceived during the interaction with spoken dialogue systems, on the basis of instrumentally or expert-derived interaction parameters. More specifically, it will be evaluated how generic model predictions are when going from one system or user group to the next. In two experiments, user quality judgments have been collected...
متن کاملProsodic, Spectral and Visual Features for the Discrimination of Prominent and Non-prominent Words
Despite its very high relevance for human communication current spoken dialog systems usually ignore the prosodic variations in the speech signal [1, 2, 3]. In [4] it was shown that speakers use prosodic cues to highlight corrections in a dialog with a machine and that these can be detected using prosodic cues. We extended this idea in [5] to the audio-visual discrimination of prominent from no...
متن کامل